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ABSTRACT 

The Louisiana Intern Teacher Assessment Program, part 
of the Louisiana Teacher Appraisal System, began with a preliminary 
field test, followed by a pilot test and implementation in 1993-94* 
Evaluation of the preliminary field test was conducted through 
responses of 223 assessors, intern teachers, principals, and assessor 
trainers to nine different questionnaires and responses in focus 
group interviews. The analysis and results of the focus groups 
interviews (conducted with 74 assessors, intern teachers, principals, 
and trainers in 10 groups) form the basis of this study. In all, 
evaluators collected 15 hours of audiotaped proceedings from the 
focus groups. Focus group participants offered solutions to the 
problems they perceived with the field test and with the research 
methodology. In general, participants felt that the system of 
appraisal had merit, although there were concerns abcut the rating 
scales used and assessor training. One table describes the focus 
groups. (Contains 5 references.) (SLD) 
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Evaluation of the Louisiana Teacher Assessment 
Program for Interns: Use of Focus Group Approach 

Introduction 

The Louisiana Intern Teaciier Assessment Program is part of the Louisiana Teacher 
Appraisal System. The Louisiana Teacher Appraisal System was endorsed by the Board of 
Elementary and Secondary Education (BESE), approved by the State Legislature and established 
with the passage of ACT 506 in 1992. The intern assessment program has three phases: (1) 
Preliminary Field Test, (2) Pilot Test and (3) Full Implementation. The Preliminary Field Test was 
conducted in the Spring of 1993. The second important phase is the full scale Pilot Test to be 
conducted during the 1993-94 school year. The Pilot Test is the final preparation for full 
implementation in 1994-94. 

The evaluation of the Preliminary Field Test of the Louisiana Intern Teacher Assessment 
Program (LTAM) was conducted by three-person assessor teams. The information collected was 
responses of assessors, intern teachers, and assessor trainers to nine (9) different questionnaires and 
the responses of assessors, intern teachers, and trainers selected to participate in focused interviews 
at the conclusion of the Field Test. 

The analysis and results of the Focus Group Interviews forms the basis of this study. It was 
necessary to triangulate data that was obtained from two of the nine questionnaires in order to '*get 
the big picture** of the participants' perceptions of the program. The components of the 
participants' perceptions of the Louisiana Intern Assessment Program included the Global 
Questionnaire, the Individual Attribute Questionnaire-Open-Ended Comments and the Focus Group 
Interviews. 

Global Questionnaires 

(ilobal Questionnaires were sent to all the participants in the Preliminary Field Test, and 
76% of them responded to the surveys. The response rate was close to or over 80% for all groups 
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except the External Assessors, who had a 58% rate of response. The overall response rate was 
certainly adequate, especially for a one-time mailout. 

Both statistical analyses of the closed-ended items and qualitative analysis of the open-ended 
items were conducted for the overall study, but the qualitative analysis performed on the open-ended 
items was used to assist in understanding and amplifying some of the focus group information. The 
qualitative analysis utilized was the constant-comparative method. 

In the constant comparative method, narrative di:ta are first unitized (the data are broken 
into distinct "units" of information), and then they are categorized (a coder assigns unique 
"categorical" names to each unit of data summarizing the essence of the written statement). 

The research questions included closed-ended and open-ended items in a number of areas 
including the following five: the Assessors' evaluation of their preparation; the Assessors' evaluation 
of the "reasonableness'* of certain actual and hypothetical aspects of the system; the Interns' 
assessment of whether or not certain mandated events actually occurred; the Assessors' perceptions 
of the strengths of the system; and the Assessors' perceptions of the weaknesses of the system. 

Individual Attribute Questionnaire - Open-Ended Responses 

Assessors and interns in the Preliminary Field Test were sent questionnaires asking them the 
difficulty of rating or demonstrating each of the 27 LTAI-I attributes. The total response rates for 
ail participants was very high - 85%. 

An item-by-item analysis of responses to the open-ended query asking for *Tomments on 
Difficulties" of the 27 attributes was conducted and utilized to obtain a better understanding and 
explanation of the Focus Group Interview responses. 

Sixty-four percent (64%) of all respondents made a comment on at least one open-ended 
item. The response rate varied widely across the items. Eight of the items (#3, 5, 6, 10, 11, 21, 23 
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and 25) received at least 25 comments. The responses to these items were used to compile the focus 
group results. 

Focus Group Interviews 

This Focus Group study was a part of the 1992-93 Preliminary Field Test of the Louisiana 
Intern Assessment Program, conducted in the Spring semester 1993. The Focus Groups were 
designed to provide complementary information to that gathered from the Global Questionnaires 
which each participant completed following the Professional Development Conferences. The Focus 
Group also provided an arena where answers were obtained to questions that couldnU be asked in 
other formats. 

The study consisted of ten Focus (Jroup seSvSions held at three sites (Jefferson, Lafayette, and 
Bossier parishes) in May 1993, The participants M^ere selected using a stratified random selection 
procedure, with the strata being participant "type" (Intern, Principal, Master Teacher, External 
Assessor) and site. Altogether there were three Intern Focus Groups, three Principal Focus Groups, 
two Master Teacher Focus Groups, and two External Assessor Focus Groups. Seventy-four (74) 
individuals participated in the Focus Groups, representing 33% of the total number of people taking 
part in the Pilot Study. 

FOCUS GROUP STUDY 

Methodology 

This section of the study details what the Focus (iroup research questions were and how they 
emerged. The Louisiana Department of Education (LDE) staff, in consultation with the LTAI-I 
development and trainin^^ team, constructed an initial list of research questions - one for the 
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Assessors and on^hr the Intern Teachers. The LDE staff then met with the Focus Group 
consuhants aniTdeveloped a revised list of questions. The two sets of revised questions are found in 
Appendix 1 (Focus Group Protocol-Assessors) and Appendix 2 (Focus Group Protocol-Interns). 
The questions for both groups were divided into the following areas: 

(1) Instrumentation and Data Collection 

(2) Post-Observation Feedback 

(3) Intern Teacher Orientation 

(4) Professional Development Plan 

(5) Merit of System 

(6) Other Comments 

Additionally, the Assessors were given a checklist concerning the knowledge and skills that they 
should receive during their training. 

A stratified random sample of Assessors and Intern Teachers were selected t^ participate in 
the Focus Group interviews. According to Krueger (1988), the "ideal focus group is composed of 
seven to ten people with similar backgrounds" (p. 93). For the current study, the participants were 
grouped homogeneously by both "type" (Intern Teacher, Principal, Master Teacher, External 
Assessor) and training area (Bossier, Jefferson, Lafayette). Krueger also concluded that the first two 
Focus Groups "provide a considerable amount of new information, but by the third ... session, a fair 
amount may have already been covered" (p.97). 

The design for the current study called for ten homogeneous Focus Groups with the 
following makeup: 

(1) Three Intern Teacher Focus Groups (one in each traininp. area); 

(2) Three Principal Focus (Jroups (one in each training area); 

(3) Two Master Teacher Focus (vroups; and 
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(4) Two External Assessor Focus Groups. 
LDE staff wanted more Information on the Intern Teacher and Principal groups; therefore, three 
Focus Groups were held for these two types of participants. 

Altogether, 136 of the 223 total participants (61%) in the Pilot Study were invited to 
participate in the ten Focus Groups. Krueger suggested that researchers over-recruit by 25% to 
compensate for "no-shows." Thus 12-14 participants were solicited for each of the Focus Groups. 

As indicated in Table 1, 74 of the 136 solicited participants 



Table 1 
Participation Rates in 
Focus Groups, by Assessor 'Type" 



Assessor 
"Type" 


Location 


No. of 
Participants 


Number Invited 


Participation 
Rate 


Intern 


Jefferson 


9 


14 


64% 


Intern 


Lafayette 


11* 


14 


79% 


Intern 


Bossier 


6 


14 


43% 


Principal 


Jefferson 


5 


14 


36% 


Principal 


Lafayette 


6 


14 


43% 


Principal 


Bossier 


5 


12 


42% 


Master Teacher 


Lafayette 


9 


14 


64% 


Master Teacher 


Bossier 


7 


12 


58% 


External 
Assessor 


Jefferson 


11 


14 


79% 


External 
Assessor 


Lafayette 


5 


14 


36% 


Totals 


10 (Groups 


74 


136 


54% 



Nine came on date of Focus (Jroup; two others were interviewed later. 
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(54% of those invited) showed up for their Focus Group session. This averaged to 7.4 participants 
per Focus Group, which is within the desirable range (7-10) described by Krueger. The participation 
varied by participant ''type": 62% of both the invited Intern Teachers and the invited Master 
Teachers came to their sessions; 57% of the External Assessors came; but only 40% of the invited 
Principals came to their sessions. All three Principal Focus (iroups had fewer than seven 
participants. 

The reasons for the low participation by the Principals include the following: 

(1) No moneta ry incentive - Krueger (1988) suggested that incentives should be given to 
Focus Group participants as a "symbol that the researcher is giving something in exchange for what 
has been received'* (p.IOO). Unfortunately, there were no funds available for incentives in the Pilot 
Study. 

(2) Bad time of the year - Due to the time constraints under which the entire Pilot Study 
took place, it was impossible to hold the Focus Groups until the end of the school year. This caused 
severe problems for several of the Principals (especially in the Bossier Parish area), who first agreed 
to participate and then pulled out at the last minute when it was impossible for the LDE to recruit 
new participants. 

(3) Scheduling errors - In one case (Jefferson Parish), there was miscommunJcation between 
the LDE staff (which made all the contacts with participants) and the Principals with regard to the 
site of the Focus Group. This contributed to the low turnout in that area. 

Despite these problems, the Principal Focus Groups were successful experiences, according 
to the interviewers. Since three groups were recruited, the researchers were able to get 4.5 hours 
of interviews (1.5 hours per session), and they indicated that the Principal Focus Groups were 
"active'* despite their low participation rates. 

It should also be noted that 30% (16 of 53) of aU Principals in the Pilot Study participated 



6 

8 



in the Focus Groups, and that these participants proportionately represented the training areas 
throughout the state. This Principal sample for the Focus Group study is, thus, certainly adequate. 
Incidentally, around 33% (74 of 223) of ajl Pilot Study participants took part in the ten Focus 
Groups. 

The participants were greeted by the Focus Group team (which in most cases consisted of 
two individuals). After sign-in and travel forms were completed, the purposes for the Focus Group 
were described. Salient points that were made included: 

(1) The value of the Focus Group methodology was explained. 

(2) The impartiality of the Focus Group team was re-iterated (i.e., the two team members 
were from an independent evaluation company, representing neither the LDE or the LTAM 
development team). 

(3) It was noted that consensus of opinion was not sought; instead a variety of opinions were 
solicited. 

(4) It was re-iterated that next year's study would also be a Pilot, and that the results from 
this year's study would be used to 'Tme tune*' the procedure for 1993-94. 

It was then explained that one team member would conduct <the meeting, while the other 
concentrated on taking notes.The purpose for audiotape recording the session was then explained, 
and assurances of confidentiality of individual responses were made. Any discomfort with the tape 
recording was then discussed, and the participants were told that if they wanted the tape recorder 
turned off at any time, the session moderator would do so. In fact, several of the participants asked 
for the tape recorder to be turned off at certain points (such as during the discussion of specific 
participants or the trainers), and the moderator complied immediately with those requests. 

In all cases, the Focus (Jroup sessions ranged from one hour twenty minutes to one hour 
forty minutes. The great majority of all the time in the sessions was audiotaped, Altogether, the 



evaluators collected 15 hours of audiotaped proceedings from the Focus (iroups sessions. Audiotapes 
and typed copies of the notes taken by the two interviewers were compiled for each of the sessions. 

There were six sets of questions asked the Focus Group participants: one set concerned with 
instrumentation and data collection, which typically took about one-half of the session to complete; 
a second short set asking questions about POSTobservation feedback; third and fourth sets 
concerning the Intern Teacher orientation and the Assessor training respectively (together with 
checklists); a fifth set composed of a question regarding the merit of the system; and a sivth set 
asking for additional comments and concerns. 

As described by Krueger (1988), there is a specific procedure for analyzing Focus Group 
results. This procedure is different from the constant comparative method of Lincoln and Guba 
(1985), which was used to analy/e the Global Questionnaire data. The constant comparative method 
involves breaking each bit of narrative data into the smallest discrete bit of information and then 
categorizing that piece using an exhaustive and elaborate coding scheme. 

With Focus (iroup data, Krueger suggests a more global process, including the following 

steps: 

(1) Reading all the summaries at one sitting and making notes of potential trends and 
patterns. 

(2) Listening to the audiotapes concentrating on one issue or question at a time. 

(3) Considering the specific context of the Focus (Jroup and the influence that context may 
have had on the group processes. 

(4) Considering the internal consistency of the responses. 

(5) Finding the "big*' ideas. (Krueger, 1988, pp.ll4-II6). 
Analysis of the Focus (iroup Study 

In analyzing the Focus Group data, we (the two Focus Group interviewers) followed this 
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procedure. We also performed aspects of the analysis separately and then came to agreement on 
important issues in Joint meetings. The consistency of responses across the Focus (Jroups aided us 
considerably in coming to consensus. 

The Focus (Jroup data, which is available upon request from, the evaluation team, include: 

(1) Typed summaries from all ten Focus Groups. These summaries merged the written notes 
of the two interviewers and are arranged in the order of the questions on the protocol. These 
summaries are approximately 27 single-spaced pages long. 

(2) Fifteen hours of audiotapes from the 10 one and one-half hour Focus Group interviews. 

(3) Checklists completed by the participants concerning their needs for further training. (Sec 
Appendices 3 & 4 for the Checklist forms.) 

With regard to the overall project, we first considered the individual responses to the Global 
Questionnaires, and then examined the evidence from the Focus Group interviews. In general, there 
was great agreement between the problems identified in the Global Questionnaires and those 
discussed in the Focus CJroups. The Focus (Jroups had an extra dimension, however, since the 
participants were asked to come up with potential solutions to the problems. This report, while 
restating the identified problems, will also present the potential solutions that the groups discussed. 

Results from the Focus (ilroup Study 

The results from the Focus Groups will be organized around the research questions identified 
in the first section of this study. Within each research question section, responses will be presented 
by type of participant (Intern Teacher, Principal, Master Teacher, External Assessor). Emphases 
will be placed not only on the identification of problems (which often coincided with those presented 
in the (Jlobal Questionnaire Report) but also on the potential solutions that the Focus Groups 
developed. The participants were eager to offer solutions to the problems that they perceived, and 
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this often led to lively and productive sessions. 

The questions concerning Instrumentation and data collection focused on ways to improve 
the eleven different forms and the techniques for gathering data using them. Most participants 
thought there were too many instruments and that it took too (ong io complete them. The 
instruments were repetitious, according to the Assessors, who made numerous suggestions on 
methods for shortening or eliminating them. For instance, the Assessors suggested combining 
question sets #5 and 6 on the PREobservation Conference/Interview Record, because they both 
concerned activities chosen to accomplish objectives in the classroom. Similarly, several Assessors 
felt that question sets #8, 9, 10, and 1 1 on that same instrument could be combined in some manner, 
since they were all concerr ^*d with student assessment and feedback. 

Almost all participants felt that the hardest attributes for Interns to demonstrate were: (1) 
identifying and accommodating individual differences in students; and (2) teaching higher order 
thinking skills (HOTS). Both the Assessors and the Interns felt that these attributes should be 
stressed more in the training and that there should be in-services for those Interns deficient on these 
attributes. 

There was some controversy over the types of rating scales employed: on the interview rating 
form, there were sliding three-point scales; cm the observation rating forms, there were fixed three- 
point scales. Most Assessors and Interns believed strongly that the scales should be consistent. The 
majority of Assessors and Interns preferred two-point scales to the three-point scales used in the 
Pilot Study, but there was a sizeable minority who thought the three-point scales were better. 
Similarly, most participants preferred the fixed point scales, although a number of individuals 
believed the sliding scales were better. Another point of concern regarding the observation rating 
forms was the use of the category ''not observed" - several Interns and Assessors were not sure 
when and how it should be coded. 
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There were four different forms associated with the Observation Record, and many Assessors 
wanted some of them eliminated. For instance, almost all Assessors agreed with eliminating the 
Observation Supplement, which was cumbersome and for which there had been little training. There 
were several suggestions for improving the Observation Scripting Form, including eliminating it 
altogether. A consistent refrain throughout all Focus Groups concerned eliminating or combining 
three forms which seemed highly repetitive: the PREobservation Conference/Interview Record, the 
Teacher Observation Analysis and Scoring Summary, and the Intern Summary Report (Individual 
Assessor Rating Form). 

Additionally, the Assessors objected to completing the same information repeatedly (their 
name; the Intern's name; their SSN; the Intern's SSN; etc.). They suggested that the state 
Department of Education precode aH the forms with this repetitive information and place it in the 
Intern Teacher's packet sent to the Assessors. 

The POSTobservation Conrerences were especially liked by the Interns in the training areas 
where they received both individual and consensus ratings from the Assessors. (In one training area 
the Interns received consensus ratings only, while in the other two areas they received both types 
of feedback.) The Interns liked the "Teacher Comments" block on the POSTobservation Conference 
Record. 

The participai s also found the Professional Development Conference to be useful, although 
a number of Interns thought the Assessors didn't take enough time to develop the PDP. The form 
itself was too short, according to many of the participants, although several liked the fact that it was 
one page long. One Assessor suggested that the form be made one page legal size. 

Most of the Assessors complained about the Assessor Training ManuaK stating that it was 
organized in a confusing manner, lacked a Table of Contents, and had instruments spread 
throughout it. They suggested some practical ways to improve the manual: color code it; put all the 
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forms in one area of the Manual (such as in the Appendix); include a detailed Table of Contents; 
include better definitions of the components and attributes; and note plainly on the forms which of 
them go to the Intern and which do not. 

With regard to the POSTobservation feedback, most of the participants preferred the 
feedback system that was utilized in their training area with two exceptions: the Interns preferred 
the individual plus consensus feedback system because it gave them "maximum" information; and 
the two External Assessor Focus Groups (one in each of the two feedback systems) both preferred 
consensus only feedback, contending that individual feedback could lead to strained relationships 
at the school site. The majority of ail participants preferred the individual plus consensus feedback 
system, but a sizeable minority preferred consensus only. Those preferring the dual feedback system 
said that it better fostered the "improvement" goals of the program, because the Interns got more 
information with which to work. 

Comments on the Intern Orientation indicated that it was not long enough, and that the 
Interns were often not prepared for the PREobservation interview. Many Assessors wanted the 
Interns to be exposed to the same videotapes and discussions that they were, so that their 
"expectations" would be similar. As might be expected, the participants selected identification and 
accommodation of individual student differences and HOTS as the two areas in which the Interns 
needed the most training. 

Comments on the Assessor Training were the most uniformly negative of any made on the 

system as a whole. As one Assessor put it, the negative training experiences "turned ofP* many 

people to the system . The Assessors' criticisms included the following: 

the training was too *' rushed"; more examples and clarification would have been 
useful; the Observation Supplement wasn't covered adequately; the trainers told 
them that they had to "justify" their ratings and the Principals didn't like this; 
training on the relative meanings of the "1","2" and "3" ratings was not adequate; 
and the trainers were not able to rationalize why they chose a "2" or a "3" when the 
Assessors disagreed with them. The Principals said that the afternoon sessions were 
disorgani/.ed and that there was "too much theory and not enough 'hands-on.'" 
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Additional comments concerned the fact that some of the trainers were from out-of-state. The 
Principals, especially, did not like that, asserting that "we have sufficient resources in Louisiana: 
we don't need experts from Tennessee." The Principals contended that the Louisiana trainers did 
a much better job than their colleagues from out-of-state. These perceptions are particularly 
disquieting since the STAR system was also criticized for having the instrument development team 
serve as trainers. 

The question asking for the merits of the system elicited responses that gave a more balanced 
picture of the participants' overall perception of it. Almost all participants felt that a system of this 
type had merit, although several also said that their District intern evaluation system already served 
this purpose. The new system was seen as much more positive than the previous one due to: its 
emphasis on "improvement" of the Intern through mentoring; the opening of channels of 
communication among the three Assessors, which was seen as an important by-product of the 
system; the greater manageability of this new set of attributes, which several Assessors said that a 
"good" teacher should be doing anyway; and the perception that the system is "fair." 

The final comments introduced a few different concerns: all participants thought the 24 hour 
rule was inoperable, suggesting 48 or 72 hours instead; several Assessors were concerned that the 
system would turn into a "dog-and-pony" show, although they didn't have many practical methods 
for avoiding that, outside of "showing up" at the Intern Teacher's class unannounced : most 
Principals felt that much of the burden of the new system fell on their shoulders, and they were 
particularly adamant about "simplifying" the paperwork and scheduling; many Master Teachers 
wanted to be the Mentor Teacher at their school and the Master Teacher (Assessor) at another 
school to avoid biased ratings and strained relationships; the Principals and Master Teachers were 
concerned about how they would get compensated for substitutes and release time; the External 
Assessors were concerned about the time required to perform the assessments, and many felt that 
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coJIege professors would not do such labor intensive work unless . aey were compensated; this latter 
point might not be such a problem since many of the Interns and other Assessors preferred District 
staff to play the role of the External Assessors anyway. 
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